{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build with RichPromptTemplate\n",
    "\n",
    "Introduced in `llama-index-core==0.12.27`, `RichPromptTemplate` is a new prompt template that allows you to build prompts with rich formatting using Jinja syntax.\n",
    "\n",
    "Using this, you can build:\n",
    "- basic prompts with variables\n",
    "- chat prompt templates in a single string\n",
    "- prompts that accept text, images, and audio\n",
    "- advanced prompts that loop or parse objects\n",
    "- and more!\n",
    "\n",
    "Let's look at some examples.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Prompt with Variables\n",
    "\n",
    "In `RichPromptTemplate`, you can use the `{{ }}` syntax to insert variables into your prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.prompts import RichPromptTemplate\n",
    "\n",
    "prompt = RichPromptTemplate(\"Hello, {{ name }}!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can format the prompt into either a string or list of chat messages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello, John!\n"
     ]
    }
   ],
   "source": [
    "print(prompt.format(name=\"John\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ChatMessage(role=<MessageRole.USER: 'user'>, additional_kwargs={}, blocks=[TextBlock(block_type='text', text='Hello, John!')])]\n"
     ]
    }
   ],
   "source": [
    "print(prompt.format_messages(name=\"John\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chat Prompt Templates\n",
    "\n",
    "You can also define chat message blocks directly in the prompt template."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = RichPromptTemplate(\n",
    "    \"\"\"\n",
    "{% chat role=\"system\" %}\n",
    "You are now chatting with {{ user }}\n",
    "{% endchat %}\n",
    "\n",
    "{% chat role=\"user\" %}\n",
    "{{ user_msg }}\n",
    "{% endchat %}\n",
    "\"\"\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ChatMessage(role=<MessageRole.SYSTEM: 'system'>, additional_kwargs={}, blocks=[TextBlock(block_type='text', text='You are now chatting with John')]), ChatMessage(role=<MessageRole.USER: 'user'>, additional_kwargs={}, blocks=[TextBlock(block_type='text', text='Hello!')])]\n"
     ]
    }
   ],
   "source": [
    "print(prompt.format_messages(user=\"John\", user_msg=\"Hello!\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prompts with Images and Audio\n",
    "\n",
    "Assuming the LLM you are using supports it, you can also include images and audio in your prompts!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://cdn.pixabay.com/photo/2016/07/07/16/46/dice-1502706_640.jpg -O image.png"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAI\n",
    "\n",
    "llm = OpenAI(model=\"gpt-4o-mini\", api_key=\"sk-...\")\n",
    "\n",
    "prompt = RichPromptTemplate(\n",
    "    \"\"\"\n",
    "Describe the following image:\n",
    "{{ image_path | image}}\n",
    "\"\"\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The image features three white dice with black dots, captured in a monochrome setting. The dice are positioned on a checkered surface, which appears to be a wooden board. The background is blurred, creating a sense of depth, while the focus remains on the dice. The overall composition emphasizes the randomness and chance associated with rolling dice.\n"
     ]
    }
   ],
   "source": [
    "messages = prompt.format_messages(image_path=\"./image.png\")\n",
    "response = llm.chat(messages)\n",
    "print(response.message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Audio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget AUDIO_URL = \"https://science.nasa.gov/wp-content/uploads/2024/04/sounds-of-mars-one-small-step-earth.wav\" -O audio.wav"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = RichPromptTemplate(\n",
    "    \"\"\"\n",
    "Describe the following audio:\n",
    "{{ audio_path | audio }}\n",
    "\"\"\"\n",
    ")\n",
    "messages = prompt.format_messages(audio_path=\"./audio.wav\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The audio features a famous quote, \"That's one small step for man, one giant leap for mankind.\" This statement was made during a significant historical event, symbolizing a monumental achievement for humanity.\n"
     ]
    }
   ],
   "source": [
    "llm = OpenAI(model=\"gpt-4o-audio-preview\", api_key=\"sk-...\")\n",
    "response = llm.chat(messages)\n",
    "print(response.message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Advanced] Loops and Objects\n",
    "\n",
    "Now, we can take this a step further. Lets assume we have a list of images and text that we want to include in our prompt.\n",
    "\n",
    "We can use the `{% for x in y %}` loop syntax to loop through the list and include the images and text in our prompt.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text_and_images = [\n",
    "    (\"This is a test\", \"./image.png\"),\n",
    "    (\"This is another test\", \"./image.png\"),\n",
    "]\n",
    "\n",
    "prompt = RichPromptTemplate(\n",
    "    \"\"\"\n",
    "{% for text, image_path in text_and_images %}\n",
    "Here is some text:\n",
    "{{ text }}\n",
    "Here is an image:\n",
    "{{ image_path | image }}\n",
    "{% endfor %}\n",
    "\"\"\"\n",
    ")\n",
    "\n",
    "messages = prompt.format_messages(text_and_images=text_and_images)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets inspect the messages to see what we have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "user\n",
      "block_type='text' text='Here is some text:'\n",
      "block_type='text' text='This is a test'\n",
      "block_type='text' text='Here is an image:'\n",
      "block_type='image' image=None path=None url=AnyUrl('\n",
      "block_type='text' text='Here is some text:'\n",
      "block_type='text' text='This is another test'\n",
      "block_type='text' text='Here is an image:'\n",
      "block_type='image' image=None path=None url=AnyUrl('\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for message in messages:\n",
    "    print(message.role.value)\n",
    "    for block in message.blocks:\n",
    "        print(str(block)[:100])\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, we have a single message with a list of blocks, each representing a new block of content (text or image).\n",
    "\n",
    "(Note: the images are resolved as base64 encoded strings when rendering the prompt)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llama-index-caVs7DDe-py3.10",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
